Link management systems and methods for multi-stage, high-speed systems

ABSTRACT

A system includes a plurality of modules, a backplane communicatively coupled to the plurality of modules, a plurality of links defined between the plurality of modules over the backplane, and a link management system configured to dynamically manage parameters associated with each of the plurality of links. A link management method includes, for a system, defining a codebook for each module, device, and interconnect in the system, the codebook includes data describing physical link topologies and configuration parameters associated therewith, for initializing a link in the system, obtaining appropriate codebooks for each segment in the link, calculating an overall link loss for the link based on data in the appropriate codebooks, and obtaining configuration parameters for the link based on the overall link loss.

FIELD OF THE INVENTION

Generally, the field of art of the present disclosure pertains tohigh-speed networking devices, and more particularly, to link managementsystems and methods for multi-stage, high-speed systems such asswitches, routers, and the like.

BACKGROUND OF THE INVENTION

Conventionally, communication networks have ever increasing demands forhigher data rates, faster switching times, and reduced operationalexpenses. For example, communication networks may be realized throughinterconnected network elements. The network elements includeingress/egress ports for various services along with switching and/orprocessing elements. The capacity of the network elements has scaledfrom 10 Gb/s to significantly more than 1 Tb/s in a single bay/frame ofequipment. The same is true for supercomputers, data centers, cloudcomputing, and the like. In physically realizing a network element,supercomputer, etc. there are a plurality of modules with variousinterconnections therebetween via high-speed links. Conventionally, thelinks include an optimized set of configuration data that is applied toall the links statically during initialization. Each link may includeone or more physical electrical link segments that may or may not bepresent all the time. The physical and electrical characteristics ofthese links vary widely at very high data speeds. However, a limited andstatic set of configuration parameters are used for all the links in thesystem. Conventional link initialization is a tedious process which iserror prone, requires frequent changes (e.g., whenever there is ahardware change), includes no systematic turn up or shut down procedure,and the like. Further, conventional modes of operation also waste poweras unused links are initialized in a same manner as used links thereforeare always powered up regardless of use.

BRIEF SUMMARY OF THE INVENTION

In an exemplary embodiment, a system includes a plurality of modules, abackplane communicatively coupled to the plurality of modules, aplurality of links defined between the plurality of modules over thebackplane, and a link management system configured to dynamically manageparameters associated with each of the plurality of links. The linkmanagement system may be configured to selectively disable any inactivelinks of the plurality of links. The system may further include aplurality of codebooks each of which is associated with one of theplurality of modules and the backplane, wherein the plurality ofcodebooks include a set of files describing physical link topologies andconfiguration parameters associated therewith. The system may furtherinclude local storage on each of the plurality of modules, whereincodebooks associated with each of the plurality of modules are stored inthe local storage. The system may further include a release file systemassociated with the system, wherein codebooks associated with thebackplane are included in the release file system.

The link management system may further include a link manager componentimplemented on at least one of the plurality of modules and configuredto manage data retrieval from the plurality of codebooks and determinethe parameters therefrom, and a link servicer component implemented onremaining modules of the plurality of modules and configured to accesslocal storage for an associated codebook of the plurality of codebooksand handle requests from the link manager component. The link managercomponent may be configured to, for a link of the plurality of links,obtain data from the plurality of codebooks on a per segment basis,compute an overall loss for the link based on the data per segment, andobtain the parameters based on the computed overall loss. The linkmanager component may be configured to continually monitor the pluralityof links, and recalculate the parameters on any of the plurality oflinks based on any monitored changes. The link manager component may beconfigured to selectively disable any inactive links of the plurality oflinks. The link manager component may be configured to push theparameters to the link servicer component for implementation thereof.The link servicer component may be configured to set transmission powerfor the link based on the pushed parameters. Each of the plurality oflinks may include an electrical link defined by at least one segment,connectors disposed at segment boundaries, and transceivers at each end.

In another exemplary embodiment, a link management method includes, fora system, defining a codebook for each module, device, and interconnectin the system, the codebook including data describing physical linktopologies and configuration parameters associated therewith; forinitializing a link in the system, obtaining appropriate codebooks foreach segment in the link; calculating an overall link loss for the linkbased on data in the appropriate codebooks; and obtaining configurationparameters for the link based on the overall link loss. The linkmanagement method may further include, for provisioned links in thesystem, monitoring link attributes, and selectively disabling inactivelinks based on the monitoring. The link management method may furtherinclude, for provisioned links in the system, monitoring link attributesfor external and internal stimuli, and dynamically updating theconfiguration parameters of any of the provisioned links in the systembased on any monitored external and internal stimuli. The linkmanagement method may further include providing a new module for thesystem, wherein the new module includes local storage with an associatedcodebook, and utilizing the new module in the calculating with the datafrom the associated codebook on the new module.

In yet another exemplary embodiment, a method of operating a high speedsystem includes operating a system including a plurality of circuitsinterconnected therein by a plurality of segments; for a new linkbetween two circuits of the plurality of circuits, wherein the new linkincludes at least one segment of the plurality of segments, obtainingdata for each of the at least one segment, performing a calculationbased on the obtained data, and determining configuration parameters forthe new link based on the calculation; for inactive segments of theplurality of segments, selectively disabling the inactive segmentsreducing power consumption of the system; and for any active segments ofthe plurality of segments, monitoring attributes associated therewithand dynamically updating configuration of any of the active segmentsbased on any monitored external and internal stimuli. The method mayfurther include, for a new module introduced subsequent to the systemoperating, obtaining data for segments using the new module from localstorage on the new module. The obtaining data may include obtaining dataassociated with segments on modules from local storage associatedtherewith, and obtaining data associated with segments on backplanelinks from a database associated with a software release for the system.

BRIEF DESCRIPTION OF THE DRAWING(S)

Exemplary and non-limiting embodiments of the present disclosure areillustrated and described herein with reference to various drawings, inwhich like reference numbers denote like method steps and/or systemcomponents, respectively, and in which:

FIG. 1 is a diagram of a switching and/or processing system with aplurality of modules disposed therein;

FIG. 2 is a diagram of a link segment in the system of FIG. 1 betweentwo modules interconnected over a backplane and connectors disposedthereon;

FIG. 3 is a flowchart of a link management method;

FIG. 4 is a diagram of a link management system associated with thesystem of FIG. 1 for implementing link management systems and methods;

FIG. 5 is a diagram of a system three modules included therein operatingthe link management system of FIG. 4;

FIG. 6 is a diagram of the system of FIG. 5 illustrating exemplarycodebooks for the link management system;

FIGS. 7 and 8 are diagrams of exemplary link topologies codebooks fromFIG. 6 for line module topology codebooks (FIG. 7) and a switch moduletopology codebook (FIG. 8);

FIGS. 9-12 are diagrams of an exemplary codebook operation in the systemof FIG. 5; and

FIGS. 13-14 are diagrams of another exemplary codebook operation in thesystem of FIG. 5.

DETAILED DESCRIPTION OF THE INVENTION

In various exemplary embodiments, link management systems and methodsare described within multi-stage, high-speed systems such as switches,routers, and the like. In an exemplary embodiment, the link managementsystems and methods are configured to calculate optimized configurationdata for links based on preconfigured parameters stored in so-calledcodebooks, a presence or absence of one or more stages or modules, atype of hardware used at each stage, etc. The link management systemsand methods also allow a deterministic order of bringing links up ordown and handling run time changes to the links such as when differentstages are removed or fail. The link management systems and methods mayreevaluate on a real time basis all the above mentioned parameters toadjusts the links, if required. That is, the link management systems andmethods make the process of link management dynamic, under the controlof software/firmware, and triggered by external and internal stimuli.

The link management systems and methods assist in running links at amuch higher speeds than previously possible leading to a largerswitching and/or processing platforms on a similar footprint. The linkmanagement systems and methods provides a scalable and deterministic wayto introduce new hardware in the field without requiring softwareupgrades, even though different link equalization parameters are needed.The link management systems and methods also help in reducing powerconsumption by keeping links powered down when they are not needed. Thelink management systems and methods decouple link initialization actionsfrom the various link parameters and use a software algorithm tocalculate and feed the link initialization parameters to the linkinitialization engine. Advantageously, the link management systems andmethods future proof the switching and/or processing systems allowingintroduction of new modules with different link characteristics in thefield.

The link management systems and methods provide an algorithmic techniquein managing equalization settings over high speed links. As describedherein, the link management systems and methods provide techniques toobtain appropriate data (e.g., through the codebooks), to calculateoverall links attributes (e.g., overall link loss), and to determineappropriate configuration parameters based thereon (e.g., equalizationsettings such as transmit equalization, transmit power levels, receiversettings, etc.). Additionally, the link management systems and methodsfocus on minimizing incremental effort whenever any part of a signalpath is modified (e.g., on associated modules, backplanes, etc. Forexample, changes may occur when new designs (e.g., new modules with newprinted circuit board (PCB)) are introduced (e.g., to provide increasedfunctionality, new services, etc.). Further, changes may occur whencurrent designs are modified or changed (e.g., a same module introducedwith a new PCB dielectric material that alters associated electricalcharacteristics. The link management systems and methods are configuredto minimize these associated changes.

As high speed systems grow to Tb/s speeds and beyond, a simple devicedriver approach does not work as proper equalization settings requiremore global information as opposed to only local information. Further,high speed systems are growing to thousands of electrical interconnects(i.e., links) contained therein. As such, hard coding every possiblelink topology is impractical. This approach was implemented in lowerspeed systems with fewer links. However, Tb/s+ systems have thousandsand thousands of interconnects with differential pairs on a star wiredbackplane. For a high speed switch system, the number of table entriesrequired in the hard coded implementation grows as a function of P*Q*Rwhere P is the number of line module types, Q is the number of backplanelinks times the number of backplane types, and R is the number of switchmodule types. Such a table can grow very large particularly if the tableneeds to accommodate intermediate vintages of product produced duringthe development process. The operation of older systems at lower speedsmakes the link performance less sensitive to changes in the channelcharacteristic. For example, some systems, such as the Gb/s systems, donot have any transmit equalization settings at all.

Another approach may include using embedded hardware, a so-called backchannel, and an embedded algorithm to allow transmit equalization to beadjusted dynamically either just at start up or possibly also on anongoing basis. A backplane version of 10 gigabit Ethernet, 10 GBase-KR,provides support for this approach by including hooks in anauto-negotiation protocol to permit an algorithm running at the receiverto communicate adjustments to the transmitter to adjust equalization atstart up. However, this approach burdens the circuit hardware withadditional logic on every port to support the dynamic adjustment. Intypical implementations, including 10 GBase-KR, these approaches requirethat the pairing of receive and transmit signal directions be maintainedthroughout the product platform. This pairing constraint can result inthe need for increased signal routing layers most typically in theswitch module routing which is undesirable due to impacts on cost,manufacturability, and long term PCB reliability.

In view of the foregoing, the link management systems and methodsprovide an algorithmic technique. Generally, a premise of the linkmanagement systems and methods is that some relatively simple parameterscharacterizing different pieces of the electrical channel can becombined algorithmically to determine the overall equalization settingswithout needing to perform analysis or calculations on a fullycomprehensive electrical model of the channel. Thus, the link managementsystems and methods are an approximate technique avoiding an approachusing a more complete model that would require large amounts of storageand be grossly burdensome for implementation on typical controlprocessors.

After some analysis, it was determined that for the channels in a highspeed system, a parameter characterizing the loss at a single criticalfrequency (f_(faud)/2) could be combined using a simple algorithm(summing) to determine the amount of equalization required and then toindex into a simple, small table mapping the amount of equalization totap settings for the transmit equalizer as well as providing anyrequired adjustments for non-adaptive portions of the receiver. In anexemplary embodiment, the link management systems and methods includeelectrical channel characteristic data stored locally on each module innon-volatile storage. Note that the proposed methodology also enablestransparent interoperability between different serializer-deserializer(SERDES) devices at each end of the link as each module is responsiblefor programming its own SERDES considering only parametric datadescribing the channel between the two devices.

The link management systems and methods are applicable to ultra highspeed and complex electronic systems, i.e. overall system capacity ofTb/s+, individual link capacity in excess of 10 Gb/s, and thousands ofsegments forming links therein. In an exemplary embodiment, the linkmanagement systems and methods may apply to serial electrical data ratesover 10 Gb/s, e.g. about 12 Gb/s. The link management systems andmethods may further also apply to even greater data rates (e.g., 18-20Gb/s serially) through increasing a number of frequency points at whichloss is specified, e.g. from one data point to two data points.

Referring to FIG. 1, in an exemplary embodiment, a diagram illustrates asystem 10 with a plurality of modules 12 disposed therein. The system 10may further include a frame 14 with interconnects through which theplurality of modules 12 interconnect therebetween. The modules 12 may bereferred to as line cards, blades, boards, plug-ins, and the like. Theinterconnects may include a backplane, a mid plane, and the like.Overall, the system 10 is configured to perform set of functions withdifferent modules 12 utilized to realize the functions. For example, thesystem 10 may include a switch, a router, a cross-connect, amultiplexer, an aggregation platform, a supercomputer node, a server, anetwork element, etc. Of note, the system 10 includes a plurality ofdifferent modules 12 each with a plurality of links interconnecting oneanother in the system 10. In an exemplary embodiment, the system 10 isconfigured to process and/or switch data in excess of 1 Tb/s. Thus,internal to the system 10, there may be significant lengths ofelectrical interconnects within and between the modules 12. That is, thesystem 10 may include internal link interconnect complexity thereinequivalent to a 1 Tb/s+ network. Further, in operation, the system 10may be configured to operate with any number of modules 12 with themodules 12 being configurable in any number of slots or locations in theframe 14. Also, the system 10 may include empty slots that areunequipped during operation.

In an exemplary embodiment, the system 10 may include an optical switchthat can consolidate the functionality of a multi-service provisioningplatform (MSPP), digital cross connect (DCS), Ethernet and OpticalTransport Network (OTN) switch, dense wave division multiplexing (DWDM)platform, etc. into a single, high-capacity intelligent switching systemproviding Layer 0, 1, and 2 consolidation. In another exemplaryembodiment, the system 10 may include any of an OTN/SONET/SDH add/dropmultiplexer, a multi-service provisioning platform (MSPP), a digitalcross-connect (DCS), an optical cross-connect, a WDM/DWDM terminal, aswitch, a router, and the like. That is, the system 10 may be referredto as a multistage switching network or a multistage processing network.Various architectures exists for switching systems, such as a Closswitch which partitions a large crossbar into a number of stages, thusreducing the complexity of the system 10 by decreasing the number ofcross-points. In the following description, the link management systemsand methods are described with reference to the system 10 being ahigh-speed switching system using a Clos switch in an exemplaryembodiment. Those of ordinary skill in the art will recognize that thelink management systems and methods contemplate use with the system 10being any type of device. That is, the link management systems andmethods may be implemented or used with any high-speed device on linkinterconnects disposed therein.

Referring to FIG. 2, in an exemplary embodiment, a link segment 20illustrates two modules 12A. 12B interconnected over a backplane 22. Ina Clos switch or any other switch, different switching stages areconnected to one another via high speed serial data segments 24, 26, 28.Note, the segments 24, 26, 28 collectively form a link 30 between themodules 12A, 12B. Each link 30 in the link segment 20 and the system 10is logically divided into n segments 24, 26, 28 where n>0 in each stage.The number of segments 24, 26, 28 may be platform dependent and may bebased on a minimum granularity needed. For example, in FIG. 2, the link30 is divided into the three segments 24, 26, 28. The segment 24 (L1) isbetween an ingress stage device (e.g., the module 12A) to a connector 32on the backplane 22. The segment 26 (L2) is on the backplane 22 betweenthe connector 32 and another connector 34 associated with the module12B. Finally, the segment 28 (L3) is between the connector 34 to themodule 12B. In this example, a user could very well choose to divide thesegments 24, 26, 28 into any other number (i.e., instead of the threedepicted herein). The segment granularity all depends on the level ofcontrol that is needed to configure the links.

Of note, each of the segments 24, 26, 28 have some loss as signalstravel thereon on. The loss may depend on length, printed circuit board(PCB) material, vias, device parastics, and several other factors. Thelink 30 (and all other links in the system 10) are initialized during apower-up sequence. Two ends 36, 38 of the data link (e.g., Transmitterand Receiver) are each initialized using initialization parameters. Theinitialization parameters needed for the link 30 may be hardwaredependent and may include link loss, link polarity (i.e., inverted ornot), maximum supported serial data rate, etc. Only using theinitialization parameters works as long as the link 30 is at arelatively moderate speed. However, when the link 30 is run at very highspeeds, the link 30 become more sensitive to link length (e.g., PCBtraces, vias, etc.), connector types, PCB material, etc. Thus, itbecomes more and more difficult to find the right set of initializationparameters that can be used to power up the links.

In various exemplary embodiment, the link management systems and methodsare configured to calculate optimized configuration data for the link 30(and other links in the system 10) based on preconfigured parameters, apresence or absence of one or more stages or modules, a type of hardwareused at each stage, etc. The link management systems and methods alsoallow a deterministic order of bringing links up or down and handlingrun time changes to the links such as when different stages are removedor fail. The link management systems and methods may reevaluate on areal time basis all the above mentioned parameters to adjusts the links,if required. That is, the link management systems and methods make theprocess of link management dynamic, under the control ofsoftware/firmware, and triggered by external and internal stimuli.

Referring to FIG. 3, in an exemplary embodiment, a flowchart illustratesa link management method 40. In an exemplary embodiment, the linkmanagement method 40 is implemented in the modules 12, in a controllermodule, in firmware, etc. of the system 10. In particular, the linkmanagement method 40 is configured to calculate optimized configurationdata for any link in the system 10. First, given a system of interestsuch as the system 10, an equalizer codebook is defined for each module,device, etc. associated with the system (step 42). The link managementmethod 40 utilizes the appropriate equalizer codebook for each segmentto calculate transmit equalization and other device specific transmitand receive parameters. That is, the codebooks include data for anoverall link calculation. In the example of FIG. 2, there are three linksegments 24, 26, 28 so there should be three sets of codebooks with dataavailable corresponding to each of the link segments 24, 26, 28. Asdescribed herein, the codebooks may include text files in CSV(comma-separated values) format that contain configuration informationwhich is used to enable datapath links on devices.

For a link in the system of interest, the link management method 40obtains the appropriate codebooks for each segment in the link (step44). That is, the link management method 40 needs to plug in the rightcodebook data for each segment. This may include a two step lookupprocess. First, modules, devices, etc. with each segment may be queriedto determine device type and an associated revision so that the rightcodebook data can be retrieved from a database. The database can eitherbe packaged with a software install load or available in a well definedlocation that a user can provision. If the first step fails to find thecodebook data, then the second step is to look for the codebook data ina local storage for the module, device, etc. The local storage could aon-board storage like a flash, non-volatile memory, a disc, etc. Thistwo step approach has one key advantage in allowing the link managementmethod 40 to override the codebook data for modules that are alreadydeployed in the field without requiring any equipment reprogramming orreturns. If it is found that there is better set of codebook data thatcould be used on a particular module, device, etc., then new softwarecan be released with the codebook packaged with the software release.The two step lookup process will then use the codebook data that camewith the software release.

With the appropriate codebooks obtained, the link management method 40calculates link loss and other link parameters for configuration thereof(step 46). Here, the link management method 40 is configured to use datafrom the codebooks to calculate an overall link loss based on thesegments, a maximum data rate of the link based on the segments, linkpolarity, and the like. With links provisioned in the system, the linkmanagement method 40 may also be configured to monitor active links forkey information such as a presence indicator (i.e., is a module in aslot or not) and a failure state of all of the modules in the variousactive links (step 48). In an exemplary embodiment, the link managementmethod 40 may selectively disable transmitters and receivers on modulesparticipating in any active links that are taken down (e.g., via afailure, via removing a module, etc.). For example in FIG. 2, if themodule 12B is removed, the link management method 40 may immediatelydisable the transmitter and receiver on the module 12A that has theother end of the link 30. Of note, this functionality of the linkmanagement method 40 of disabling inactive links advantageously savessignificantly power consumption. For example, if the system 10 has 30modules 12 and only 10 are equipped, i.e. 20 are unequipped, then fortylinks can be disabled reducing power consumption by more than 200 W.

Referring to FIG. 4, in an exemplary embodiment, a diagram illustrates alink management system 50 associated with the system 10 and/or themodules 12 for implementing the link management systems and methods. Forexample, the link management system 50 may be configured to implementthe link management method 40. In the exemplary embodiment of FIG. 4,the link management system 50 is illustrated with a controller module12C and three switching modules 12D. Those of ordinary skill in the artwill recognize the link management system 50 may include any number ofmodules 12. Also, the system 10 may include any number of differenttypes of modules 12. For example, one type of module may include thecontroller module 12C which may be configured to control and monitorfunctionality of the system 10, e.g. operations, administration,maintenance, and provisioning (OAM&P). Other exemplary types of modulesmay include the switching modules 12D, line modules, and the like. Thecontroller module 12C includes a codebook 52A and the switching modules12D each include a codebook 52B.

The link management system 50 includes a link manager software component54 and a link servicer software component 56. The link management system50 is configured to parses link topology and device configuration datafrom the codebooks 52, drive a link enable/disable process on moduleinsertion/removal/HIR (hold in reset), and drive the link enable/disableprocess for module internal components on mode changes. Generally, thelink manager software component 54 is a controller-side process thatparses backplane topology and manages links between the modules 12 andmanages all backplane-facing datapath links. The link servicer softwarecomponent 56 is a module-side process that parses module internaltopology and manages links between internal components on the module 12.

In an exemplary embodiment, the link manager software component 54 isdeployed on the controller module 12C. If the controller module 12C isnot available or in other exemplary embodiments, the link managersoftware component 54 may be deployed on any of the modules 12. The linkmanager software component 54 is configured to host software componentsthat manage nodal for the link management systems and methods. Forexample, the link manager software component 54 may be configured toimplement/control the link management method 40. Also, the controllermodules 12C may also include nodal environment software 58. The software58 is configured to control and operate all aspects of the system 10 andthe functionality associated with each of the modules 12. For example,the software 58 may be the software install load with the database forcodebook data.

The link manager software component 54 is configured to load a commonbackplane codebook from the codebook 52A, and load module states (e.g.,present, failed) for every module 12 in the system 10. In an exemplaryembodiment, the module states may be determined from the nodalenvironment software 58 which may monitor module status for OAM&P. If anew link is needed in the system 10, the link manager software component54 may be configured to implement the link management method 40. Forexample, the link manager software component 54 can initiate a codebookdata retrieval from each of the modules 12D. This may include eitherretrieving data from the codebooks 52B on the modules 12D, retrievingmodule type, revision number, etc. and retrieving data from the codebook52A on the module 12C, or querying the database based on the moduletype, revision number, etc.

With all of the codebook data, the link manager software component 54 isconfigured to evaluate/calculate link parameters for each link segmentsand derive the total link loss. The link manager software component 54then pushes the calculated link loss to the link servicer softwarecomponents 56 in a right order (based on the link segment topology), asrequired by the hardware specifications. Also, the link manager softwarecomponent 54 can verify that the links are enabled and publish thisinformation so that other software components can now use the links toset up connections. The link manager software component 54 also cancontinue to monitor, on real time basis, all the modules 12 andrecalculate associated link parameters for any new environment changes.Further, the link manager software component 54 can also handle anyfailures reported by any of the link servicer software components 56 andraise alarms for such failures.

The link servicer software components 56 is usually deployed on themodules 12D. That is, the link manager software component 54 may beconsidered a master on the controller module 12C, and the link servicersoftware components 56 may be considered as a slave on other modules12D. Also, the link servicer software components 56 could be deployed asa proxy on the controller module 12C. The link servicer softwarecomponents 56 are configured to access codebook data from the codebook52B via an onboard flash or from a persistent database, handle activateand deactivate links requests from the link manager software component54, and provide feedback to the link manager software component 54 forerror scenarios such as a codebook read failure, link enable/disablefailure, etc.

Referring to FIG. 5, in an exemplary embodiment, a diagram of a system10-1 is illustrated with three modules 12-1, 12-2, 12-3 included thereinoperating the link management system 50. The system 10-1 is an exemplarythree-stage switch with the modules 12-1, 12-3 being line modules oringress/egress stages, and the module 12-2 being a switch module orcenter stage. The system 10-1 may include a controller link manager 60and a line module link manager 62 with the requisite area of controlover the modules 12 illustrated in FIG. 5. The controller link manager60 may include the link manager software component 54 and all of theassociated functionality. The line module link manager 62 may includethe link servicer software components 56 and all of the associatedfunctionality. The modules 12-1, 12-3 include ports 64 and switchingcircuitry 66. The controller link manager 60 includes software classeswhich parse, validate, and cache codebook data. The controller linkmanager 60 further includes a module state machine which drives theaction based on module state transitions. The line module link manager62 includes software classes which parse, validate, and cache internalcodebook data for the line modules 12-1, 12-3. The line module linkmanager 62 also monitors for port mode changes and then drives linkenable/disable on mode change.

The ports 64 provide physical interfaces to the system 10-1, e.g.optical, electrical, etc., and the switching circuitry 66 may include afirst/third stage switch. The modules 12-2 include switching circuitry68. As described herein, a link may be formed through the system 10-1based on a plurality of segments. A datapath through the system 10-1 maytraverse the entire three-stage architecture through the variousswitching circuitry 66, 68. As described herein, the datapath linksbetween stages may have different lengths due to module placement, PCBdesign, etc. Also as described herein, in order to initialize thedataplane for a given module, the links must be enabled withcalculated/determined configuration parameters. The configurationparameters used during link enabling/disabling are determined based ontotal path loss, and these configuration parameters configureequalization functionality on a per-link basis. The configurationparameters are provided via the codebooks.

Referring to FIG. 6, in an exemplary embodiment, a diagram of the system10-1 illustrates exemplary codebooks 70 for the link management system50. As described herein, the codebooks 70 are essentially a set of filesdescribing the physical link topology between devices on a module andlinks between modules (on the backplane). The set of files containconfiguration parameters for each device whose dataplane links will beenabled by the link management system 50. The exemplary system 10-1includes seven codebooks including internal line module codebooks 70-1,70-7; line module topology codebooks 70-2, 70-6; backplane topologycodebooks 70-3, 70-5; and a switch module topology codebook 70-4.Further, embedded within the codebooks 70-1, 70-2, 70-4, 70-6, 70-7,there may be device configuration codebooks that contain data specificto each datapath in these devices (e.g., over the circuitry 66, 68, theports 64, etc.).

In an exemplary embodiment, the backplane topology codebooks 70-3, 70-5are distributed with a release file system (e.g., the nodal environmentsoftware 58). It is expected that the backplane topology codebooks 70-3,70-5 relate to relatively unchanging equipment, i.e. the physicalbackplanes in the system 10. The codebooks 70-1, 70-2, 70-4, 70-6, 70-7may be distributed in memory stored on the physical modules 12-1, 12-2,12-3, or with the release file system (where the data is looked up basedon module type, revision number, etc.). In an exemplary embodiment, thememory stored on the physical modules 12-1, 12-2, 12-3 includes flashmemory, non-volatile memory, etc. Additionally, the link managementsystem 50 may include an override/update mechanism to allow codebook 70updates to existing hardware. For example, the link management system 50may obtain the codebooks 70 from either the release file system or thememory on the physical modules 12-1, 12-2, 12-3. Here, the linkmanagement system 50 may verify that the codebook 70 in the memory isthe same as a similar codebook in the release file system. If not, thelink management system 50 may utilize the most up to date version (whichis likely part of the release file system) and/or update the memoryaccordingly. In another exemplary embodiment, the link management system50 may be configured, upon a software update of the release file system,to update any of the codebooks 70 in the memory as needed.

Referring to FIGS. 7 and 8, in an exemplary embodiment, diagramsillustrate exemplary link topologies for the line module topologycodebooks 70-2, 70-6 (FIG. 7) and the switch module topology codebook70-4 (FIG. 8). The codebook link topology refers to a physical layout oflinks on a board/circuit. In an exemplary embodiment, the links may bereferred to by a logical or physical number. The codebooks 70 provideenough information that a logical-to-physical mapping can be created bythe link management system 50. FIG. 7 illustrates the codebook 70-2 forthe module 12-1. The module 12-1 includes circuitry 66-1, 66-2 acting asingress and egress switches. Each of the circuitry 66-1, 66-2 hassegments 72 to/from other modules such as the module 12-2 or otherswitch modules. In an exemplary embodiment, each of the circuitry 66-1,66-2 includes three segments to each separate switch module (SM0 . . .SM8). FIG. 8 illustrates the codebook 70-4 for the module 12-2. Themodule 12-2 includes circuitry 68-1, 68-2, 68-3 each acting as a centerstage switch. Each of the circuitry 68-1, 68-2, 68-3 has segments 74from line modules (such as the modules 12-1, 12-3) and segments 76 tothe line modules (such as the modules 12-1, 12-3). In an exemplaryembodiment, the system 10-1 may support 30 line modules each with threeconnections per switch module as shown in FIG. 8. As noted herein, FIGS.5-8 are presented herein for illustration purposes, and the linkmanagement systems and methods contemplate operation on any type ofmodule 12 with any type of links and/or segments therebetween.

Referring to FIGS. 9-12, in an exemplary embodiment, an exemplarycodebook operation in the system 10-1 is illustrated using the linkmanagement system 50 and the controller link manager 60 to determineparameters for a link 90. FIG. 9 illustrates a logical diagram of thelink 90 between the circuitry 66-1 on the module 12-1 to the circuitry68-1 on the module 12-2. FIGS. 10-12 illustrate physical diagrams of thelink 90 and associated segments 100, 102, 104. The link 90 includes thethree segments 100, 102, 104, and the link management system 50 utilizesthe codebooks 70-2, 70-3, 70-4 to look up losses and calculate a totalloss for the link 90 followed by a look up of the appropriateconfiguration parameters based on the total loss.

FIG. 10 illustrates the system 10-1 with the codebook 70-3. For example,the following table is an example of the codebook 70-3 for the BackplaneLink Topology.

# Backplane Link Topology Table # START OF METADATA TYPE   1 REV   1NROW 1620 HEAD TXCard TXSlot TXLink RXCard RXSlot RXLink Fabric TypeLossFIG. MIN LM 1 1 LM 1 1 DATA 0 MAX SM 30 90 SM 30 90 TDM 100 . . .DATA LM 3 1 SM 1 7 TDM 59 DATA LM 3 2 SM 1 8 TDM 59 DATA LM 3 3 SM 1 9TDM 59 DATA LM 3 4 SM 2 7 TDM 59 DATA LM 3 5 SM 2 8 TDM 59 DATA LM 3 6SM 2 9 TDM 59 DATA LM 3 7 SM 3 7 TDM 63 DATA LM 3 8 SM 3 8 TDM 63 . . .

The codebook 70-3 may include a type indicator (indicative of a type ofcodebook), a revision (REV) indicator (indicative of whether thecodebook 70-3 is current), a number of rows indicator (NROW) fordetermining a size of the codebook 70-3, and data rows. The data rowsmay include a first two rows which are a minimum and maximum value forthe codebook 70-3 (in this case, slots=min of 1, max of 30; links=min of1, max of 90; a fabric type, e.g., DATA, TDM, etc.; and loss figures=minof 0, max of 100). In this example, the codebook operation is concernedwith a segment 100 between TXLink 6 of the module 12-1 and RXLink 9 ofthe module 12-2, and per the highlighted section of the codebook 70-3,this segment includes a loss figure of 59.

FIG. 11 illustrates the system 10-1 with the codebook 70-4. For example,the following table is an example of the codebook 70-4 for the switchmodule topology.

# Switch Module Topology Table # START OF METADATA TYPE  2 REV  2 NROW180 HEAD Direction Xcvr Link InvPolar MaxData LossFIG. XcvrNo XcvrPortMIN EGRESS FE 1 0 1562 0 1 1 MAX INGRESS STSX 90 1 6520 100 3 96 . . .DATA INGRESS STSX 1 1 6520 36 2 11 DATA INGRESS STSX 2 1 6520 38 1 11DATA INGRESS STSX 3 0 6520 37 3 3 DATA INGRESS STSX 4 1 6520 37 1 10DATA INGRESS STSX 5 0 6520 36 3 4 DATA INGRESS STSX 6 0 6520 40 2 2 DATAINGRESS STSX 7 0 6520 34 3 8 DATA INGRESS STSX 8 1 6520 39 2 12 DATAINGRESS STSX 9 1 6520 37 1 12 DATA INGRESS STSX 10 1 6520 40 2 16

The codebook 70-4 may include a type indicator (indicative of a type ofcodebook), a revision (REV) indicator (indicative of whether thecodebook 70-4 is current), a number of rows indicator (NROW) fordetermining a size of the codebook 70-4, and data rows. The data rowsmay include a first two rows which are a minimum and maximum value forthe codebook 70-4 (in this case, direction=egress or ingress,transceiver (Xcvr)=FE or STSX, link=1 . . . 90, Inverse Polarity=0 . . .1, max data rate=1562 . . . 6520, loss figure=0 . . . 100, transceivernumber=1 . . . 3, and transceiver port=1 . . . 96). In this example, thecodebook operation is concerned with a segment 102 on link 9 totransceiver port 12, and per the highlighted section of the codebook70-4, this segment includes a loss figure of 37.

FIG. 12 illustrates the system 10-1 with the codebook 70-2. For example,the following table is an example of the codebook 70-2 for the linemodule topology.

# Line Module Topology Table # START OF METADATA TYPE  3 REV  1 NROW 64HEAD Direction Xcvr Link InvPolar MaxData LossFIG. XcvrNo XcvrPort MINEGRESS FE 1 0 6520 0 1 1 MAX INGRESS STSX 32 1 6520 100 2 32 . . . DATAINGRESS STSX 1 0 6520 12 1 3 DATA INGRESS STSX 2 0 6520 13 1 4 DATAINGRESS STSX 3 0 6520 13 1 5 DATA INGRESS STSX 4 0 6520 13 1 8 DATAINGRESS STSX 5 0 6520 12 1 2 DATA INGRESS STSX 6 0 6520 11 1 1 DATAINGRESS STSX 7 0 6520 12 1 6

The codebook 70-2 may include a type indicator (indicative of a type ofcodebook), a revision (REV) indicator (indicative of whether thecodebook 70-2 is current), a number of rows indicator (NROW) fordetermining a size of the codebook 70-2, and data rows. The data rowsmay include a first two rows which are a minimum and maximum value forthe codebook 70-4 (in this case, direction=egress or ingress,transceiver (Xcvr)=FE or STSX, link=1 . . . 32, Inverse Polarity=0 . . .1, max data rate=6520 . . . 6520, loss figure=0 . . . 100, transceivernumber=1 . . . 2, and transceiver port=1 . . . 32). In this example, thecodebook operation is concerned with a segment 104 on link 6 totransceiver port 1, and per the codebook 70-2, this segment includes aloss figure of 11.

With the segments 100, 102, 104 determined, a total loss value isdetermined for the link, and a loss index may be calculated to identifya set of link enable parameters. For example, total loss=107=59+37+11. Aformula for the loss index may be equal to floor((totalloss)/4)=floor(107/4)=floor(26.75)=26, so 26 may be a loss index forfinding the correct link enable parameters. The following tableillustrates a configuration table where the loss index may be utilizedto look up the correct link enable parameters.

# Configuration Table # START OF METADATA TYPE  2 REV  1 NROW 35 HEADLossIndex FFEMode C0 C1 C2 C3 TXPow SlewRt DFEMode . . . MIN 10 0 −15  0−31 −15 0 0 0 MAX 44 3 15 63  31 15 127 1 3 # LossIndex describes totalchannel loss in units of 0.4 dB . . . DATA 23 1 −9 63 −17 0 74 0 0 DATA24 1 −9 63 −18 0 76 0 0 DATA 25 1 −10 63 −18 0 77 0 0 DATA 26 1 −10 63−18 0 78 0 0 DATA 27 1 −10 63 −19 0 82 0 0

With the loss index of 26, the operation now has the appropriateconfiguration settings for the link of the segments 100, 102, 104.

Referring to FIGS. 13 and 14, in an exemplary embodiment, an exemplarycodebook operation in the system 10-1 is illustrated using the linkmanagement system 50 and the controller link manager 60 to determineparameters for a link 110. In this example, the codebook 70-1 for theinternal line module topology is used. FIG. 13 illustrates a logicaldiagram of the link 110, and FIG. 14 illustrates a physical diagram ofthe link 110. The link 110 may be from the circuitry 66-2 on the module12-1 to a field programmable gate array 112 associated with a port. Forexample, the following table is an example of the codebook 70-1 for theinternal line module topology.

# Line Module Internal Topology Table # START OF METADATA TYPE  7 REV  1NROW 160 # CONFIG FILE SerDesType TTYP TREV DREV CONFIG_DATA STSX 8 1 1CONFIG_DATA FPGA1 11  1 2 CONFIG_DATA FPGA2 12  1 2 # FIELD METADATAHEAD DeviceName PortGrp Dir SerDesPt SerDesType . . . ConfigIndex DATAEGRESS_STSX 1 TX 1 STSX 6 DATA MAPA_FPGA 1 RX 15  FPGA1 2

The last two data rows include all information needed to find the properconfiguration values for the link 110. Note that in the codebook 70-1, aTX/RX pair for a given link is uniquely identified by its Interface(here EGRESS_STSX-MAPA_FGPA), and the like. These two rows indicate thatthe logical link in question (FIG. 13) originates from port 1 on theSTSX and terminates on port 15 of the MAPA_FPGA. So, the physical linkin question is illustrated in FIG. 14. Having found the rows pertainingto the link 110, this data contained therein is used to look up theactual link configuration parameters. On the second to last row, it isshown that the SerDesType given is STSX. Near the top of the file, it isshown a CONFIG_DATA line whose CONFIG_FILE is STSX. This line indicatesthat the configuration file for the STSX SerDesType has a TTYP (TableType) of 8, TREV (Table Revision) of 1, and DREV (Data Revision) of 1.With this information, and referring back to the second to last row, theConfiglndex for the STSX side of the link is 6. A configuration table isillustrated below for the STSX side.

# STSX Configuration Table # START OF METADATA TYPE 8 TREV 1 NROW 15HEAD ConfigIndex FFEMode C0 C1 C2 C3 TXPow SlewRt DFEMode . . . MIN 0 0−15  0 −31 −15 0 0 0 MAX 14 3 15 63 31 15 127 1 3 # START OF DATA DATA 41 −1 63 −8 0 86 0 0 DATA 5 1 −2 63 −9 0 86 0 0 DATA 6 1 −3 63 −11 0 86 00 DATA 7 1 −3 63 −12 0 86 0 0 DATA 8 1 −4 63 −15 0 86 0 0

The highlighted row for the Configlndex of 6 provides the parameters forenabling the egress link of the circuitry 66-2 (STSX). Similarly for theFPGA1, whose CONFIG_DATA gave a TTYP of 11, TREV of 1, and DREV of 2.The desired Configlndex is 2 and the configuration table is illustratedbelow for the FPGA1 side with the highlighted row for the Configlndex of2 providing the parameters for enabling the link of the FPGA 112.

# FPGA1 Configuration Table # START OF METADATA TYPE 11 TREV 1 NROW 16HEAD ConfigIndex TxDiffCtl TXPostEmp TXPreEmp RXEqMix . . . MIN 0 0 0 00 MAX 20 15 31 15 7 # START OF DATA DATA 0 7 0 0 3 DATA 1 7 6 1 3 DATA 28 9 1 1 DATA 3 8 12 2 1 DATA 4 8 14 2 7

It will be appreciated that some exemplary embodiments described hereinmay include one or more generic or specialized processors (or“processing devices”) such as microprocessors, digital signalprocessors, customized processors and field programmable gate arrays(FPGAs) and unique stored program instructions (including both softwareand firmware) that control the one or more processors to implement, inconjunction with certain non-processor circuits, some, most, or all ofthe functions of the methods and/or systems described herein.Alternatively, some or all functions may be implemented by a statemachine that has no stored program instructions, or in one or moreapplication specific integrated circuits (ASICs), in which each functionor some combinations of certain of the functions are implemented ascustom logic. Of course, a combination of the two approaches may beused. Moreover, some exemplary embodiments may be implemented as acomputer-readable storage medium having computer readable code storedthereon for programming a computer, server, appliance, device, etc. eachof which may include a processor to perform methods as described andclaimed herein. Examples of such computer-readable storage mediumsinclude, but are not limited to, a hard disk, an optical storage device,a magnetic storage device, a ROM (Read Only Memory), a PROM(Programmable Read Only Memory), an EPROM (Erasable Programmable ReadOnly Memory), an EEPROM (Electrically Erasable Programmable Read OnlyMemory), a Flash memory, and the like.

Although the present disclosure has been illustrated and describedherein with reference to preferred embodiments and specific examplesthereof, it will be readily apparent to those of ordinary skill in theart that other embodiments and examples may perform similar functionsand/or achieve like results. All such equivalent embodiments andexamples are within the spirit and scope of the present disclosure andare intended to be covered by the following claims.

What is claimed is:
 1. A system, comprising: a plurality of modules; abackplane communicatively coupled to the plurality of modules; aplurality of links over the backplane defined between the plurality ofmodules wherein each of the plurality of links comprises a first segmentof electrical connection on one of the plurality of modules, a pluralityof one or more electrical connections over traces on the backplane, anda second segment of electrical connection on another of the plurality ofmodules, and wherein lengths of the plurality of links are dependentupon locations of the plurality of modules on the backplane; and a linkmanagement system to dynamically managing equalization parameterscomprising transmit equalization, transmit power levels, and receiversettings associated with the plurality of links based on an associatedlength, the link management system to selectively disabling any inactivelinks of the plurality of links to any of a removed module of theplurality of modules or an unequipped module of the plurality ofmodules.
 2. The system of claim 1, further comprising: a plurality ofcodebooks each of which is associated with one of the plurality ofmodules and the backplane, wherein the plurality of codebooks comprise aset of files describing physical link topologies and configurationparameters associated therewith.
 3. The system of claim 2, furthercomprising: local storage on each of the plurality of modules, whereincodebooks associated with each of the plurality of modules are stored inthe local storage.
 4. The system of claim 2, further comprising: arelease file system associated with the system, wherein codebooksassociated with the backplane are included in the release file system.5. The system of claim 2, wherein the link management system comprises:a link manager component implemented on at least one of the plurality ofmodules and managing data retrieval from the plurality of codebooks anddetermine the equalization parameters therefrom; and a link servicercomponent implemented on remaining modules of the plurality of modulesand accessing local storage for an associated codebook of the pluralityof codebooks and handle requests from the link manager component.
 6. Thesystem of claim 5, wherein the link manager component is to: for a linkof the plurality of links, obtain data from the plurality of codebookson a per segment basis; compute an overall loss for the link based onthe data per segment; and obtain the parameters based on the computedoverall loss.
 7. The system of claim 5, wherein the link managercomponent is to: continually monitor the plurality of links; andrecalculate the parameters on any of the plurality of links based on anymonitored changes.
 8. The system of claim 5, wherein the link managercomponent is to: selectively disable any inactive links of the pluralityof links.
 9. The system of claim 6, wherein the link manager componentis to: push the equalization parameters to the link servicer componentfor implementation thereof.
 10. The system of claim 9, wherein the linkservicer component is to: set transmission power for the link based onthe pushed equalization parameters.
 11. The system of claim 1, whereineach of the plurality of links comprises an electrical link defined byat least one segment, connectors disposed at segment boundaries, andtransceivers at each end.
 12. A link management method, comprising: fora system, defining a codebook for each module, device, and interconnectin the system, the codebook comprising data describing physical linktopologies and configuration parameters associated therewith; forinitializing a link in the system, obtaining codebooks for each segmentin the link; calculating an overall link loss for the link based on datain the codebooks; and obtaining configuration parameters for the linkbased on the overall link loss, wherein the link comprises a firstsegment of electrical connection on a module, an electrical connectionover traces on a backplane, and a second segment of electricalconnection on another module, and wherein a length of the link isdependent upon locations of the modules on the backplane, and whereinany inactive links of the plurality of links are selectively disableddue to any of a removed module of the plurality of modules or anunequipped module of the plurality of modules.
 13. The link managementmethod of claim 12, further comprising: for provisioned links in thesystem, monitoring link attributes; and selectively disabling inactivelinks based on the monitoring.
 14. The link management method of claim12, further comprising: for provisioned links in the system, monitoringlink attributes for external and internal stimuli; and dynamicallyupdating the configuration parameters of any of the provisioned links inthe system based on any monitored external and internal stimuli.
 15. Thelink management method of claim 12, further comprising: providing a newmodule for the system, wherein the new module comprises local storagewith an associated codebook; and utilizing the new module in thecalculating with the data from the associated codebook on the newmodule.
 16. A method of operating a high speed system, comprising:operating a system comprising a plurality of circuits on differentmodules interconnected by a plurality of segments; for a new linkbetween two circuits of the plurality of circuits, the new linkcomprising at least one segment of the plurality of segments, obtainingdata for each of the at least one segment, performing a calculationbased on the obtained data, and determining configuration parameters forthe new link based on the calculation; for inactive segments of theplurality of segments, selectively disabling the inactive segmentsthereby reducing power consumption of the system; and for any activesegments of the plurality of segments, monitoring attributes associatedtherewith and dynamically updating configuration of any of the activesegments based on any monitored external and internal stimuli, whereinthe new link comprises an electrical connection between the two circuitsof the plurality of circuits over traces on a backplane and wherein alength of the new link is dependent upon locations of the two circuitsof the plurality of circuits on the backplane, and wherein any inactivesegments of the plurality of segments are selectively disabled due toany of a removed circuit of the plurality of circuits or an unequippedcircuit of the plurality of circuits.
 17. The method of claim 16,further comprising: for a new module introduced subsequent to the systemoperating, obtaining data for segments using the new module from localstorage on the new module.
 18. The method of claim 16, wherein theobtaining data comprises: obtaining data associated with segments onmodules from local storage associated therewith; and obtaining dataassociated with segments on backplane links from a database associatedwith a software release for the system.